Look at a product on a shelf, tap the temple of a pair of smart glasses, and hear a spoken investment thesis, the parent company, a supplier, and the non-obvious pick-and-shovel play, read back through the glasses' own speakers before you've put the item down.
Context
I built FinSight for FCAT, the Fidelity Center for Applied Technology, to answer one question I kept coming back to as a designer: what is a pair of smart glasses actually good for, once the novelty wears off? Every glasses demo I'd seen still routed the interesting part back through a phone screen. I wanted to test a stricter version of the idea, a post-screen interface where the primary output is spoken, the input is a glance and a tap, and the phone only comes out if you actually want to read something.
Problem
Meta's Wearables Device Access Toolkit gives you a live camera stream and a way to react to a temple tap. It does not give you a photo sync API, a way to ask "which images in this person's Photos library came from the glasses." Bluetooth bandwidth rules out a high-resolution stream. And the entire premise only holds up if the output feels natural spoken aloud, not like a screen's worth of text read out one word at a time. Every one of those constraints forced a real decision, not a default.
Decisions
Meta's SDK has no sync API for glasses photos, so once a picture lands in the iOS Photos library there is no field that says "this came from the glasses." 1Decision 01Meta's Wearables Device Access Toolkit has no sync API for glasses photos, so once a picture lands in the iOS Photos library there's no built-in way to tell it apart from every other photo on the phone. Chose: I read EXIF and TIFF maker metadata instead, matching Make, Model, and LensMake strings against meta and ray-ban, with metadata-only reads so nothing triggers a full-image decode or an iCloud fetch, plus a three-tier cache so a 180-day scan of the library stays cheap. Traded: If Meta ever changes what strings the glasses write into a photo's metadata, this detection quietly breaks, and there's no API contract that would warn me. The glasses stream over Bluetooth, which is the real bottleneck between the hardware and the phone, and a full-resolution preview would stutter before a temple tap could even register. 2Decision 02A full-resolution live preview over Bluetooth would lag or drop the connection before a temple tap could register as a capture trigger. Constraint: The SDK exposes a pause event as the closest thing to a dedicated shutter API, not a purpose-built capture call. Chose: I stream at a deliberately low 360x640 at 15fps and treat the SDK's pause event as the capture signal, keeping the connection light enough that a tap responds instantly. Traded: Every photo the app analyzes starts out lower-resolution than a normal phone photo, so a distant or cropped shelf label can be harder for GPT-4o to read. The whole point of testing glasses as an interface is that the phone screen stops being the primary surface, so the output had to be built to be heard, not read. 3Decision 03A person glancing at a shelf and tapping a temple isn't going to pull out their phone and read a paragraph, so a screen-first summary would defeat the premise of the whole project. Chose: I write the model's summary to be spoken, not read, route it to the glasses' speakers through OpenAI TTS with an Apple synthesizer fallback, and let voice commands like 'save this' file an insight straight into a Notebook without the phone ever coming out of a pocket. Traded: Deciding what earns a spoken sentence versus what only shows up on the phone screen is a judgment call I make by hand for every screen, there's no rule enforcing it. Left alone, GPT-4o defaults to the obvious answer for any product, the parent brand, which is the one idea everyone already has. 4Decision 04Asking GPT-4o for investment ideas about a product on a shelf defaults to the parent brand, the least useful insight because it's the one everyone already knows. Chose: I wrote the prompt with hard quotas: 5 to 7 insights per scan, at least 4 carrying a real ticker, and mandatory coverage across a Parent company, a Supplier, and a Pick-and-Shovel or ETF play, with a worked example built in, TSMC as the pick-and-shovel behind a phone, roughly 90 percent of advanced-chip share, fabs costing over 20 billion dollars, so the model has a template for non-obvious instead of a vague instruction. Traded: Quotas are a blunt tool, on a product with a genuinely thin supply chain the model can still be reaching to fill a slot, and nothing automatically checks that a forced insight is actually a good one.
Result
The core loop works end to end: look, tap, listen. Tickers get live quotes through Fidelity's own fastquote service, price, day change, sector, market cap, P/E, so a spoken thesis is backed by a real number, not a guess. A session ID paired with a speech-generation counter means a newer scan can't get talked over by an older one still finishing its sentence. A demo mode runs the whole pipeline from the Photos library alone, no glasses required, which is how I tested most of this from a desk. Five screens hold the rest of the app: Scans, a 3-column gallery of what the glasses have captured, Watchlist, Notebook, a Concepts glossary that grows automatically as the model names financial terms out loud, and Setup. Sign in with Apple and Firestore keep a person's data synced across devices, and the OpenAI API key lives in the Keychain, not in a config file.
It is an honest work in progress. The build spans 9 commits from December 2025 to February 2026, and a rebrand from the working title Investment Finder to FinSight is sitting uncommitted in the working tree right now. There's no App Store release, no outside testers yet, and no claim here beyond what the pipeline actually does today.
- 9
- commits, December 2025 to February 2026
- 360x640 @ 15fps
- glasses stream resolution, tuned for Bluetooth bandwidth
- 5-7
- insights returned per scan, enforced by prompt quota
- 180
- day window for the EXIF provenance cache
