Quantified-Self

This is why I’m not satisfied with a Spreadsheet:

• I’m using 5-8 different trackers, and their date formats are not consistent, and also the data has to be cleaned up, and I’m not very good at working in VBA/Excel.I don’t know how to setup my data, in a way that will allow me to see correlations between variables from different trackers…I’m better at using Python, and I will probably also use Python more than VBA/Excel in my Software Engineer career I think.

This is why I’m not satisfied with exist.io, gyroscope, etc:

• They don’t allow me to import my .CSV tracking data from Chronometer and Notion.

This is my perfect solution:

• Each month I download my .CSV files and send them to my E-mail. This should trigger IFTTT to send those .CSV files to my Webapp by using my webapp’s public API (Or will this be overkill? Don’t know how hard it is to make a public API…)
• After the .CSV files are sent to my webapp, then my webapp should extract the information about all variables and put it in a consistent format that can be worked with.
• Do you think it would be best to put it in a consistent format in a database or a new .CSV file?
• After I have the data in a consistent format that can be worked with, I should be able to do the following:
• See change through time on a graph for each variable, and also put more than one variable on the same graph.
• Making scatterplots with 2 variables
• Find correlation between any 2 variables:
• The 2 variables both can be numbers or both consist of categories or one be a number variable while the other consists of categories.
• I should be able to pick a variable, and then the webapp calculates the correlation between that variable and ALL other variables and present a sorted list.
• Maybe also other useful ways of working with the data? I’m new to this whole field, so I don’t know of other useful ways of working with the data…

I’ll offer some counsel based on a decade of observing projects similar to these. The goal of having a dashboard like product to track everything has attracted dozens (perhaps even hundreds) of programmers. Sometimes these were commercial projects, sometimes they were hobbyist or community or more generally open projects. A few have survived. Zenobase, exist.io and Gyroscope are the most well known examples in the QS Community. (Q: Who am I leaving out?) In general this has proven to be much more difficult than most people realize when they begin, due to heterogeneity on both sides of the system: users have more divergent needs than is obvious from the beginning; and, data flows from services and apps are more idiosyncratic and less stable than is obvious. You have to be prepared for a long haul. Maybe you have source of income that allows you to tinker freely for years; that’s great. Or perhaps you have a startup intensity that allows you to raise money to support the development and customer acquisition phase. That can also work. Or maybe you have a specific use case you envision, a problem to solve for people who have no other solution and will pay you for your early versions and support you as you go forward incrementally. Also can work! But I encourage you to ask the question: How does this survive for, say, three years in a way that allows me to devote significant development and support time to it.

https://exploratory.openhumans.org/notebooks/

I’m in the same process as you are, building a QS dashboard using python. It’s something I have been thinking of for a while but haven’t started until a couple of weeks ago. Here are some of my thoughts on my experience so far as well as some of the questions you asked:

• START WITH A VERY BASIC MINIMUM VALUABLE PRODUCT (MVP): that’s really the most important thing I believe: draw what the most basic interface would look like for you, and get into coding it right away instead of trying to build the complete / complicated final version you have in mind. Here is my MVP for example: a simple dashboard connected to my OURA ring data, and displaying the time I went to bed over the last 7 days.

• It’s ugly (no CSS effort), its incomplete. But it is still something I DO watch everyday when I boot my PC up. 2 reasons to focus on an MVP: 1/ it’s something tangible so it motivates you to continue, 2/ you’ll already have to learn and do a lot to have even a basic thing working (chose a library for building the interface, connect data to the API, learn how to embed a graph, …)
• For the interface I’m using dash 18. Advantages: easy integration of interactive plotly graphs (it’s done for that). Drawbacks: less flexibility than other solutions like React.js (which I don’t know yet, so learning dash is easier)
• For the database: you can start with a very basic SQLite database, which is a SQL database but stored in a file (no need to install any additional DB software), then upgrade to an MySQL or an equivalent. But at the very beginning (for the mvp) just stick to csv files, then make it more robust.
• for importing your data, you could add an “upload” block direclty in your interface (drag and drop your csv) so you don’t need to go through an intermediary email. (just an idea)
• I personally am not interested in correlations for now, as I find through my experience that its a lot of noise. I’m focusing on building something that gives me a good and flexible view of the metrics I care about so that I can myself spot correlations just by watching the trends. Automated correlations are definitely not on my priority list.
• Start small, then progressively move towards your ‘100 variables through hopefully more than 50 years’ vision.

• I think many people in this QS community have cobbled together similar solutions that work for them including myself. From a solution standpoint, the big hurdle used to be the knowledge and skills to actually build something. But with the current batch of self-service analytics tools from the major tech companies, those hurdles are no longer there. This is to say that the value proposition has to be much more than solving the problem of data integration, analysis and visualization.
• There’s a screen shot missing form your post - it’s the one that demonstrates the solution’s unique ability to turn tracker data into insights that inform (i.e. beyond descriptive analytics) an individual’s quest for self-improvement (or whatever the goal may be). In your prototype, what would that screenshot look like?
• I think https://gyrosco.pe has this market nailed pretty well. They’ve been around for a while now and easiest way for non-technical folks to get started with quantified self.
• I use it because it’s easy to rationalize as a research business expense. But I don’t find it compelling enough to engage with; I only occasionally skim the email updates. From that perspective, I find exist.io a better value particularly because I believe I still meaningfully own my data there.
• I tried some platforms that give insights based on my data, but I never felt I could trust the insights, because I didn’t knew where the data was coming from and what methods they were using to extract the insights.
• I’m new to quantified self, but trying to solve the problems associated with existing trackers as well. Mainly 1) inability to add custom data and 2) clunky interfaces. Would you be interested in checking out my concept for a tracker to alleviate these issues? https://getmission.co
• For those who want to build it, here is a comprehensive article from Tarek Ibrahim - product leader of humanapi.co
• I agree that the workflows for bringing data into our own data store for analysis are underdeveloped; but, on the other hand, having seen many, many efforts in this area over the last decade I can say that I’ve learned (at least) that it’s harder than it looks. Take a look at Open Humans 18 for an example of an open approach to building some of these workflows, using a centralized, community governed data store.

I am also interested in the creation of software like this. For a couple of years, I have been tracking data about myself in a bespoke system I made for myself, to discover correlations. Only in the last weeks have I started looking at existing software, and discovered that there is a term for quantified-self and a community around it.

Of special interest is a feedback loop, where the data is used to drive behavioral changes and discovering what effects they have on the data gathered. A reason for this is that I have medical conditions that I want to remedy through changes in behavior. There is an impressive amount of existing software for this, and I’m still experimenting with them. Nothing quite scratches my itch yet, though. In particular, nothing combines quantifying self with tools for facilitating the growth of helpful habits and the withering of detrimental ones.

On these forums, I have found posts expressing things that people want from software like this, and in this thread in particular. I have extra motivation for writing this software, namely that I am about to finish my computer science degree, and I want to both develop marketable skills as a developer and engineer, as well as hone skills from my studies, like applying machine learning and AI in practice. Because of the former, I am particular about which technologies and services I would like to use to develop and deploy. However, I reckon these preferences to be a good match for creating and deploying such software.

So for I haven’t delved much into the research in the area, but plan to do so in the coming weeks.

I would very much like to cooperate with you guys who have ideas and motivation for creating The One Dashboard, and signed up for the show and tell on the 14’th. Also, if anyone wants to discuss ideas, you can reach me at thomas.stenhaug@gmail.com

Like all of us, I’m a huge fan of quantified self apps and self tracking. I am particularly big fan of Whoop for tracking my personal fitness, sleep, and recovery. It actually inspired me to build a similar product but for my work, called Rize. Rize is a productivity tracker that shows you how you spend your time at work, improves your focus, and prevents burnout. I’ve attached a screenshot of the dashboard below.

You can check it out at https://rize.io and if you use the referral code D7708A you get your first month free. No access

I posted here in this forum back in August when I first started working on Rize and got some great feedback so I thought I’d follow up since I just launched Rize publicly. Right now it’s just a macOS app but Windows and Linux are in the works.

I’d love to hear what you all think and hope you signup! Let me know if you have any questions or feedback.

Maybe we could discuss what we think existing solutions for making personal dashboards are lacking? (Exist.io, Gyroscope, Heads up health, Google Data Studio, Grafana, Tableau, etc.)

This is what I think is lacking:

• Enough integrations with trackers we use (but this problem is hard to solve because people use so many different trackers and not all have API’s)
• .CSV import
• Flexibility of data analysis
• Custom made experiments
• Good custom tracking with numbers.

I haven’t seen a solution that satisfies ALL of the above, but only some.

I pretty much 100% agree with your assessment. I personally would also add:

1. Automatic backup of data.
2. API for simple daily import of data (e.g. from manual or unsupported trackers). To me CSV import is inferior solution, especially for dashboard, where the point is to have the data visualized persistently.
3. I would expand the ‘flexible data analysis’ to following specific analysis that I feel are essential (but I haven’t seen anywhere):a. cross-correlation across time to uncover temporally delayed relationships between tracked variablesb. automatic application of the above to all pairs of variables with report of the strongest relationships foundc. proper reporting of statistical significance of any correlations foundd. multi-variate regression

It is worth pointing out that bonus of 2 is that if such feature existed, at least the technically more adept users could integrate many trackers through services like Zapier or Integromat even if they are not supported directly in the dashboard.

Here’s what we were building lately itslili.com

It’s called Lili and it’s a health data insight assistant. We are currently available only for Fitbit users, but will soon open up to Apple Health as well.

Data import:

• Integrations (Oura, RescueTime, Google Fit, fitbit, Withings, Google Calendar, Cronometer)
• Manual upload of .CSV
• Maybe also public API?

The reason I don’t use the existing solutions are:

• I track with the Android app “Track & Graph” and NOBODY supports it…
• I want to see percentage of time in each of my Calendars as a pie chart.
• Since I track so many variables I want to create multiple Dashboards.
• I want to quickly put any two variables on a Graph/Scatterplot to explore my data
• I’m looking to automate the import of data somehow

I’m currently working on app to tackle this exact issue. I haven’t been satisfied with Exist.io 10 or Gyroscope for similar reasons that were stated. The lack of customization and data sources have left me wanting more. The goal with the app is to give a lot of flexibility to create custom dashboards and to focus on power users at the early stage.

Depending on how solid the APIs are, we’d like to integrate with a bunch of health apps, productivity apps, social media, custom spreadsheets, custom databases, weather, Github (and related), note taking apps, and whatever else our users want. For dashboard widgets, we have plans to create a lot of standard chart types (line charts, bar charts, etc.), maps, markdown widgets, maps, etc… There are also spots in the architecture where we’d like to allow the users to write custom code and custom queries to really let power users get the data shaped the way they want.

We are trying to learn from other apps so that we can scale up the number of data sources and to allow for a lot of flexibility. This app is specifically being made with the Quantified Self folks in mind towards so if anyone wants to discuss, feel free to email me at jljorgenson18@gmail.com. We would LOVE to interview potential users and it would help us out a ton to see what you guys would want in a product.

Also, we aren’t certain about the timeline yet but the hope is to have an invite only version of the app ready in a few months for iOS, Android, and Web. If there is an interest, I can send a link when it’s ready.

Personally I think in order to dramatically increase the pace of technologies development at first we need to build something like humanapi but free for both sides: users (data sources) and data consumers like analytic/insight services, dashboards, and other health focused enterprises.

The main question is who is going to pay for the development and maintain expenses? We can make it open source and try crowdfunding, or find a way around to make money on additional service. I don’t know, it’s just my thoughts. What do you think guys?

As an internet product maker I wish I could be focusing on outcomes rather than developing integrations and spending my small resources on maintenance such a huge infrastructure just for using the data. Doesn’t it discourage you? Perhaps we should combine our effort to help each other.

I pretty much 100% agree with your assessment. I personally would also add:

Automatic backup of data. API for simple daily import of data (e.g. from manual or unsupported trackers). To me CSV import is inferior solution, especially for dashboard, where the point is to have the data visualized persistently. I would expand the ‘flexible data analysis’ to following specific analysis that I feel are essential (but I haven’t seen anywhere): a. cross-correlation across time to uncover temporally delayed relationships between tracked variables b. automatic application of the above to all pairs of variables with report of the strongest relationships found c. proper reporting of statistical significance of any correlations found d. multi-variate regression It is worth pointing out that bonus of 2 is that if such feature existed, at least the technically more adept users could integrate many trackers through services like Zapier or Integromat even if they are not supported directly in the dashboard.

Platform risks assessment.

There is a stable world trend - the protection of personal data.

The sale of button phones is increasing.

Probably 50% of consumers of mobile devices will be indifferent to the safety of their personal data

However, an increasing number of consumers begin to protect personal data.

Promising solutions should store all data on the user’s devices and protect them with encryption.

The use of blockchain in startups is expensive.

Rarely, anyone in this forum mentions the safety of services.

I’ve always been impressed by the tools and dashboards that Quantified Self community have built to track and understand themselves. Like many of you, I’ve also tried building my own tracking and journaling tools for my mental well-being, but I found the process of tracking to be tedious. I believe that tracking should be effortless, like using Agaricus’s one-button tracker (thread) or a device that works in the background like a FitBit.

As a designer who has worked in e-commerce, I’ve also seen firsthand how big tech companies use our data to manipulate our behavior. I started wondering if there was a way to put that vast amount of data to good use for our own well-being. This led me to this wonderful Quantified Self community and the realization that a lot of problems can be solved through understanding our own behavior.

Inspired by Kelly_Finn 's comment and Wolfram’s Personal Analytics 3, I’ve been working on a tool that allows you to store, organize, and understand your digital footprint from a single place, using data exported from apps like Facebook. The tool also offers personalized reports and dashboard to empower you to take action on your own well-being.

I would love to get your feedback and thoughts on this project, and if you’re interested, you can sign up for early access on the landing page 4. Hopefully, I’m heading in the right direction.

My friend and I recently released a free tool called Metriport 33 that’s exactly built around the concept of a ‘personal dashboard for self-tracking data’. With Metriport you can track anything, integrate with Apple Health & Google Fit, and discover correlations between the different things you’re tracking.

We’re just getting started and would love to get feedback on our product from the QS community!

I installed the Metriprt Android app, and was impressed.

The onboarding went smoothly. Obviously you care about UX, and the polishing you’ve put into this app shows!

I’m not an experienced QSer, so I can’t compare your app with other solutions in this space.

I did get confused at the very end of the onboarding sequence, when it was prompting me about Factors. I didn’t know what I was supposed to do, but eventually I clicked on a button that allowed me to continue, and finish, the onboarding. On the home screen I was able to figure out what was meant by Factors, and they look useful.

These are my first impressions, right after installing the app.

I’ve been using Metriport for the last week or so and it definitely fills a gap. Purchased the lifetime Pro upgrade and I’m looking forward to exploring the correlations after the 10-day minimum wait.

@metriport I’d love to see widgets for single numerical entries, so that I can take one screen for my android device and dedicate to logging frequent items (e.g. a single button to track number of cups of coffee etc).

I’m using the Fitbit Sense to track my sleep and I’m looking to see how I can extract my app usage data to see if I can spot any correlations between my sleep and my usage of Instagram/Twitter etc. Seems like there’s a long way to go before we get elegant interoperability between different data sources.

Thanks very much for the kind words and feedback! Also, apologies for the late reply. First time posting on this forum, so we missed all the notifications.

Noted down your comment about Factors during onboarding, we’ll make sure to look into how we can make the purpose of Factors more explicit for new users.

If you have any more feedback or anything else for us, please feel free to connect with us in the following ways:• Our community on Reddit• Our community on Element (similar to Discord, but privacy-focused)• Shooting us an email hello@metriport.ai

Very happy to hear that you are getting use out of it, and our apologies for the late reply! We missed all the notifications here as it was our first time posting.

Regarding the widgets for single numerical entries, I can see you’ve already found the items on our roadmap:

• Home screen widget 2
• “Quick Add” for numeric data entries 2

We definitely still have a long way to go with interoperability between different data integrations, but will be expanding on this significantly in the near future. Noted down the possibility of screen time on different apps, would be interesting to see how that affects sleep for sure and we’ll give this some thought. Would be great to chat regarding any further suggestions/ideas you have going forward, so we will make sure to keep in touch!

As we started out making a Quantified Self app ourselves, @celsinga and I wanted to share our new open source Health Devices API which we just launched as it may be useful to other QS devs in this community. It allows you to gain access to your users’ health data from various wearables, RPM devices, and mHealth apps. You can check out the public

Github repo

We wished an open source product like this existed when we were making our QS app, so we wanted to share it here as well. For developers in the healthtech space, building integrations to various sources of healthcare data is a huge pain, as it means wrangling different data formats, gaining access to the APIs in the first place, and ultimately spending precious developer resources building out data pipelines. We solve this pain for you through our single open source API, which is quick and easy to get started with.

Integrations

Out of the box, our Health Devices API supports the following integrations:

• Fitbit
• Oura
• Whoop
• Withings
• Cronometer

With these initial integrations, we plan to add many more to this list, including Garmin, Apple Health, and Google Fit in the near term. If there’s an integration you need that’s not currently on roadmap, we’re more than happy to add it and build it into our API (contact us at hello@metriport.ai). Because we’re also open-source, you can also just fork our code and build your own custom integrations!

Getting Started

The easiest way to get started is to follow our quickstart guide in our dev docs.

Would love to get some feedback from the QS community about the API so we can improve it, and thanks in advance!

It’s great to see an open-source competitor to Validic and Human API, this is overdue… But please tell me you’re not returning all timestamps in UTC! 🤪

much like Gyroscope’s experiments, but shared!