[DMP 2024]: Design Map Data Storage Backend

[ChakshuGautam]

Description

GeoQuery.in is aims to become India's leading free and open-source mapping tool, dedicated to offering precise and current government body geographical information accessible to everyone. Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. However, as the number of features in the map increases with scale, we need to redesign the backend to support the storage of Map Data as well as data for layers above it + features. This involves a total redesign of the backend and will be broken down into several steps.

Currently, we are storing JSON Data everywhere. With scale, the number of features in the map would increase

• Govt Buildings
• Mandis for farmers

The objective is to redesign the backend to support for storage of Map Data as as well as data for layers above it + features. This is a total redesign of the backend.

Goals & Mid-Point Milestone

• Redesign backend to support storage of Map Data.
• Redesign backend to support storage of data for layers above the map.
• Redesign backend to support storage of features.
• Ensure scalability and efficiency in storing and retrieving data.

Setup/Installation

See here.

Expected Outcome

A redesigned backend system for GeoQuery.in capable of efficiently storing Map Data, layers above it, and features, ensuring scalability and backward compatibility.

Acceptance Criteria

• Fully functional Swagger doc of all APIs
• Benchmarks for all APIs using k6

Implementation Details

• Setup persistent layer for map data
• Store and Query Map Data realtime (100ms).
• APIs for map layers (CRUD)
• Benchmarking on core APIs

Mockups/Wireframes

No response

Product Name

GeoQuery.in

Organisation Name

SamagraX

Domain

⁠Agriculture

Tech Skills Needed

JavaScript

Mentor(s)

@ChakshuGautam

Category

Backend, Other

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

[swastik-akhil]

@ChakshuGautam I am also encountering the same issue.
The file PARSED_MASTER_LOCATION_NAMES.json is not present which is causing the issue.

[swastik-akhil]

@ChakshuGautam I am also encountering the same issue. The file PARSED_MASTER_LOCATION_NAMES.json is not present which is causing the issue.

@Savio629 @ChakshuGautam Kindly help me with this one.

[Anushkabh]

@Savio629 are there any existing issues which i can solve and include in my proposal?

[35C4n0r]

@ChakshuGautam @Savio629 @dhruv-1001
Just a few questions issue: #125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis..
Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage.
Or the preference leans towards a NoSQL database

[ChakshuGautam]

@ChakshuGautam @Savio629 @dhruv-1001 Just a few questions issue: #125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis.. Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage. Or the preference leans towards a NoSQL database

• The preference is to use PostGIS.
• Mandis data can be found here - https://enam.gov.in/web/apmc-contact-details. We will have to scrape for it.
• For other Govt buildings - we would need to figure out similar sources. One that I have found is https://igod.gov.in/, which includes sectors at the bottom which has a drill down. We will have to figure out how to scrape those as well.

Although the project's goals are not to scrape - but to build APIs that would help us to store the information. Information gathering is a separate task that we should create a ticket for. Similar exercise was done for Districts and Villages earlier.

Would you (or anyone else) like to take a crack at creating a ticket? I can review and we can get started with collecting the data so that by the time DMP is completed we have a repository of all Govt related buildings too.

[ChakshuGautam]

@Ishu2412 Please share OS of your system and also which commands have you tried to run? It's difficult to debug by just seeing the image

I am using windows 10 and also I followed the command instruction from the setup link which you have provided

Can you create a separate ticket for this? We would want to support windows long term but currently prioritizing linux. Please get started with WSL. Thanks.

I am closing this comment.

[ChakshuGautam]

@Anushkabh would you like to raise a PR for documentation? I believe we are lacking on that front and your inputs would be valuable.

@Savio629 can you create a new ticket for Documentation and direct everyone there? Thanks.

[Anushkabh]

@Anushkabh would you like to raise a PR for documentation? I believe we are lacking on that front and your inputs would be valuable.

@Savio629 can you create a new ticket for Documentation and direct everyone there? Thanks.

Okay i'll do that

[35C4n0r]

@ChakshuGautam @Savio629 @dhruv-1001 Just a few questions issue: #125 when we talk about Currently, we store JSON Data for various features we are talking about the geoJSON files right? Currently, we store JSON Data for various features such as Govt Buildings and Mandis for farmers. Sorry but I'm unable to find this, afaik we are only having the geoJSON files for Indian state, district etc boundaries, and their properties don't have data for mandis.. Also is there a DB preference maintainers favor, cause the initial idea that came to mind was to use Postgres with PostGIS for spatial data storage. Or the preference leans towards a NoSQL database

• The preference is to use PostGIS.
• Mandis data can be found here - https://enam.gov.in/web/apmc-contact-details. We will have to scrape for it.
• For other Govt buildings - we would need to figure out similar sources. One that I have found is https://igod.gov.in/, which includes sectors at the bottom which has a drill down. We will have to figure out how to scrape those as well.

Although the project's goals are not to scrape - but to build APIs that would help us to store the information. Information gathering is a separate task that we should create a ticket for. Similar exercise was done for Districts and Villages earlier.

Would you (or anyone else) like to take a crack at creating a ticket? I can review and we can get started with collecting the data so that by the time DMP is completed we have a repository of all Govt related buildings too.

@ChakshuGautam I've sent you the proposal for feedback, if you find some time available, kindly take a look at it.

[li-veindark]

Hello @ChakshuGautam. Can you explain me what do you mean by "layers above the data"? Are these layers are not same as the layers in the "openStreetMap"? If not, how?