Custom Stream API Documentation

The Custom Stream API has recently undergone a complete rewrite from the ground up, using a brand new architecture that will offer users improved reliability, scalability and functionality as well as providing a platform for launching some exciting new Data Hub features and applications over the coming months. The API is now built around a MongoDB platform, allowing users to tap into many of MongoDB’s advanced querying methods. Whilst the functionality of the old stream API will be maintained, we advise users to read through this documentation and update any scripts or sensors that push or pull data via the API accordingly.

The following API functionality is currently supported:

Write

Pushing new data to a dataset is performed through a HTTP POST request, using Basic Authentication, with the dataset access key being supplied as the authentication username and the password being left blank. The URL for pushing data using the API is as follows:

https://api.mksmart.org/data/object/<dataset-uuid>/

Data should be formatted as a JSON document and passed as the HTTP request body, using application/json as a Content-Type HTTP request header. A typical JSON document to be added to a dataset might look as follows: Note that your datapoint may be given a unique ID of your choosing, by specifying the attribute _id within your document body. This is necessary, should you wish to later update or delete datapoints via the API. If the _id field is omitted, a unique ID will be assigned to your datapoint automatically.

Examples

Additional annotations to your data

Note that, as data gets added to your dataset, the API will automatically annotate your JSON documents with some additional values. These help add context to your data and can be useful when constructing queries to return specific ranges of data. These additional fields can be distinguished from your own fields as they are preceded with an underscore:
  • _datasetid: The UUID of the dataset the JSON document was added to
  • _timestamp: The UNIX timestamp indicating when the data point was added to the dataset
  • _timestamp_year: The year component of the timestamp
  • _timestamp_month: The month component of the timestamp
  • _timestamp_day: The day (of the month) component of the timestamp
  • _timestamp_hour: The hour component of the timestamp
  • _timestamp_minute: The minute component of the timestamp
  • _timestamp_second: The second component of the timestamp
Be aware that the timestamp values added to your data reflect the time at which the JSON document was added to your dataset via the API, not the time at which a sensor value, for example, was read from a sensor. This should be considered when batch processing large amounts of data and writing to your dataset in bulk.

A data point that has been annotated with additional fields may look as follows:

Read

A dataset can be queried through a HTTP POST request with Basic Authentication, with the dataset access key being supplied as the authentication username and the password being left blank. The dataset query URL is as follows:

https://api.mksmart.org/data/query/<dataset-uuid>/

Making a simple POST request to this URL, with the appropriate authentication, will return all the data points from this dataset as an array of JSON documents. Note that this request can also be made via a HTTP GET, although using GET does not support any of the advanced querying methods described below and can only be used to return all data points.

Examples

Specifying a query

A reading request can be refined using a query over the fields of data points contained in the specified dataset. The Custom Stream API uses MongoDB’s JSON-based query language for querying data. The query document should be constructed and passed in the body of the HTTP POST request, specifying application/json as a Content-Type HTTP request header.

The simplest query, an empty JSON document, will return all data points: To specify equality conditions, use <field>:<value> expressions in the query filter document: Multiple equality conditions can be specified, using commas to separate them: Query operators can be used to construct more sophisticated queries and filter on a range of conditions. For example, to select values greater than or equal to a particular value, the $gte operator can be used:

Examples

Putting the above query into practice within a script would look like the following:
MongoDB provides a sophisticated query language that offers similar functionality to that available in SQL. We recommend taking a look at the full documentation on constructing MongoDB-style query documents which is available here.

Update

Individual datapoints can be updated, using the HTTP PUT method against the following URI

https://api.mksmart.org/data/object/<dataset-uuid>/<_id>

Use Basic Authentication to authenticate the request, with the dataset access key being supplied as the authentication username and the password being left blank. Once again, as with creating the original datapoint, the new datapoint should be formatted as a JSON document and passed as the HTTP request body, using application/json as a Content-Type HTTP request header.

Note that in order to be able to update an existing datapoint, the datapoint must have been originally created with your own personal id value, specified in the original JSON using the _id attribute. Also note that the id of a datapoint cannot be changed during the update process – any new _id attribute included in the new JSON document will be ignored.

Delete

Individual datapoints can be permanently deleted, using the HTTP DELETE method against the following URI

https://api.mksmart.org/data/object/<datraset-uuid>/<_id>

Again, note that in order to be able to delete an existing datapoint, the datapoint must have been originally created with your own personal id value, specified in the original JSON using the _id attribute.