Source Documentation

Source Document

All Item documents come from a data source (a public API, a scraped web page, a static CSV file, etc). To get more information about the source assigned to an Item, you can use the /source endpoint. For example:

import requests

url = ''
r = requests.get(url)
print r.json()

Here's an example source document:

  _id: "536dc57416863a6e16229af0",
  createdAt: "2014-05-31T21:58:20.892Z",
  description: " Violation Documentation Center in Syria is an independent, civil non-profit, non-governmental organization that started its work of monitoring and documenting the violations of human rights in Syria since April 2011.",
  endDate: "2015-05-29T02:03:37.260Z",
  frequency: "repeats",
  hasRun: true,
  id: "536dc57416863a6e16229af0",
  internalID: "1e44370d-6dcc-4813-b6cc-b104beeddc22",
  isDynamic: false,
  language: "python",
  lastRun: "2014-05-31T20:20:37.778Z",
  repeatsEvery: "day",
  sourceDataType: "json",
  sourceType: "vdc_syria",
  startDate: "2014-05-07T00:00:00.000Z",
  status: "active",
  license: "cc"

Most of these properties don't impact the nature of the data retrieved from a given source, however we've outlined the properties below that might be of interest. If a property isn't listed in the table below, you can assume it's only used internally.

Property Name Description Type
createdAt Datetime at which the source created in CrisisNET. string in format %Y-%m-%dT%H:%M:%S.%f (format code explanation)
description Free-text description written by whoever wrote the code that ingests data from this source into CrisisNET. string
startDate Datetime at which CrisisNET will start retrieving information from this source. string in format %Y-%m-%dT%H:%M:%S.%f (format code explanation)
endDate Datetime at which CrisisNET will stop retrieving information from this source. string in format %Y-%m-%dT%H:%M:%S.%f (format code explanation)
frequency Either repeats or once. A source that repeats is polled on a regular basis, but a static dataset that is no longer updated at its source will be retrieved once. string
repeatsEvery How often are sources with a frequency of repeats polled? Options are minute, hour, day, week or month. string
sourceType Often unique, but many sources may be from the same larger platform (a social network like Facebook, for example). string
status Can be active, failing or inactive. Data is only retrieved from active sources. string
license References a license code that you can use to lookup additional details using the /license endpoint. string