Xively, Phant & DataStore
I have been working on a home sensor network for several months now and had been passing this data to Xively but I started to think about the next steps, namely using this data to control devices. To achieve this I would need to look at the sensor data and output something based on some rules. I like the idea of the sensors collecting the data potentially being the output nodes so I decided that the endpoint the sensors send their data to should subsequently respond with a data packet containing a variety of dynamically controlled values.
For example if the parameter saying I am at home is true and the temperature is less than 15 degrees a heater should turn on. This information is coming from several sensors so they can't make that decision themselves but every time a new value is posted to an endpoint that url should return a json array containing a variety of details all controlled by the data coming in.
For this to work I would need to move away from Xively so I went looking for alternatives. Sparkfun recently released phant which looks ideal but its a node app and I wanted something in php that I could work with, I couldn't find anything good so I decided to build my own.
It took me a few days to get the basics in place but I quickly had a system I could send data to.
The Database
I initially started out using Amazon's Dynamo for the main database, my reason for this was the price as its effectively free for my data volume. Its also modern, fast and a system I hadn't used before which made me choose it, its also not suited for this purpose at all!
Dynamo is a key value store that has guaranteed fast reads and writes, this started out great but as I built out the back end I realised I couldn't query the system properly. I looked for an alternative and quickly settled on SimpleDB, this is a much older non relational database. I had used it many years ago for storing a lot of geodata but hadn't touched it since and apparently neither had Amazon.
Amazon appear to have abandoned the service but its got no depreciation notices so I am hopeful it will be around for a while. The key difference between the services for me was the ability to perform an sql style query, this allows me to filter the data and return only what I want for displaying in tables and graphs.
The SimpleDB service has a limit of 2500 records per query which is causing some problems, a lot of the graphs have 20 to 30 thousand points which means it needs to perform up to a dozen queries. I am still looking for a way to better manage this but waiting 10 seconds for a big graph to load isn't to bad.
I had started using dynamo for storing the other types of data but this quickly became unworkable so I switched those to MySQL or specifically amazons RDS service, it was then that I discovered the huge cost savings of using reserved instances. In my case I know I will need this db for a year so I can pay a large chunk upfront and only pay a minimal amount each month.
Admin Interface
Once I had the basics in place and the data was coming in correctly I could start to build out the admin interface.
I wanted the ability to update data feeds, graphs and triggers from the interface rather than in the code as this would make everything a lot easier to manage.
The interface has 3 main areas, streams which are the data feeds, graphs which allow me to choose a feed and data item to plot and finally triggers which can be used to send push notifications based on the incoming data.
The stream pages list the raw data that is coming in and to improve the usefulness of this page I am using pusher to send the new data to the browser in real time.
Future
The system is currently up and running and working well, I am feeding in data from various sources and the graphs and triggers are working. The next step is the response data, this needs to be a configurable json array which can pull data from different sources including using some basic formulas. I use Salesforce quite a bit and as much as I hate it its ability to create formula fields which are driven from various sources is really nice so I am probably going to use this as a template.
Open Source
I have made this project open source and its available on GitHub for anyone to download and use and improve.