PathSense

Grand Winner & 1st Place Best Use of Cohere at Hack the North 2024 hackathon (September 2024).

PathSense is a voice-driven indoor navigation assistant designed to empower visually impaired individuals by providing real-time navigation and object detection. Key features include hands-free voice commands, continuous object detection, verbal scene descriptions, and precise turn-by-turn indoor routing using advanced mapping technology. PathSense adapts in real-time to user movement and environmental changes, ensuring an intuitive and responsive experience that enhances mobility and independence in complex indoor spaces.

PathSense integrates multiple advanced technologies for seamless voice-based indoor navigation. It uses Voiceflow for managing conversations and interpreting user intents. The computer vision pipeline includes Detectron for object detection, DPT for depth estimation, and GPT-4 Vision for scene analysis. Data is stored in a Convex database in JSON format, while our custom rerank API (built with Cohere) performs semantic searches on computer vision tags and mapping data. For indoor mapping and route generation, MappedIn SDK provides floor plans. Speech processing is handled by a Groq model (based on OpenAI's Whisper) for transcription and Unreal Engine for text-to-speech. Multiple TAPO cameras stream 1080p video for real-time scene capture, making the system responsive and adaptive to user queries and environmental changes.

PathSense @ Hack the North Closing Ceremony:

https://www.youtube.com/live/I5dP9mbnx4M?si=mSYiU7WKp6DLslpm&t=6264

Press: https://www.mappedin.com/resources/blog/hack-the-north-2024/