Hi,
Today, I will talk about how I resolved the pre-assignment from Solita. Honestly, this blog will solve the challenge from a beginner’s or IT student’s view. However, that’s why the solution can not be the best, and there may be several mistakes. So, if you find out something, please let me know because I am always open to learning from mistakes and trying to be better by 1% day by day 🙂 Let’s go!
Link to the assignment: https://github.com/solita/dev-academy-2023-exercise (just in case you don’t have it yet).
1) What is it?
“An interesting project offers to create a UI and a backend service for displaying data from journeys made with city bikes in the Helsinki Capital area” from Solita.
They give us three large CSV files; we need to visualize them in UI and manage them on the server.
2) Data validation
a) Validation
Firstly, I looked at these files first, and I had some general information about them.
- Too large for a student like me. (3 million rows)
- CSV files
- Main headers like departure time, return station, duration,…
Secondly, I planned the strategy to beat them down. My strategy:
- Read them like step 1, but with programming code.
- Validate by some homemade functions.
- Write validated data to the JSON files.
(I got stuck for a couple of days cause I wondered what type of file I should choose: JSON or CSV. CSV files are smaller than JSON files; however, we got a problem with special characters after importing them.)
My code here: https://github.com/HoangCaesar/Fullstack_Projects/blob/master/Helsinki_City_Bike/data_validation/validateData.js
b) Import to databases
You might have a question why did I stick to the issue above?
I was using the free databases from MongoDB, and they limited this version <= 512Mb for an account. So that’s why I thought I would run out of memories if I wrote them all into JSON files (bigger sizes).
Finally, I created four different accounts to have more space, and I chose JSON type to avoid getting into trouble with special characters.
We can import by hand (mongodb compass) or command. In my opinion, I prefer to use Mongoimport to write command lines importing data because It sounds more professional 🙂
2) Server
a) Set up
I divided this challenge into four parts: journey list, station list, highlights, and user, which are the name of controllers also, respectively.
Furthermore, I built a Error handler to manage error and log events to record history of errors.
About technologies:
“express-generator”: to generate an application and to get started with express
“nodemon”: automatically restarting the node application when file changes are detected.
“mongoose”: to work with MongoDB.
“cors”: to enable CORS with varios options.
“dotenv”: to load environment variables from a “.env” file into process.env.
“helmet”: to secure the app by setting several HTTP headers.
“http-errors”: to create HTTP errors for each request (if any).
“date-fns”: manipulating JS dates.
“bcrypt”: to hash passwords.
“cross-env”: to set environment variables.
“googleapis”: to use google api, set credentials by oAuth2
“ioredis”: to connect to redis in Nodejs
“joi”: to validate information of user.
“jsonwebtoken”: to create access token and refresh token.
“nodemailer: to send verify email to user.
*********************** Dev Dependencies *****************************
“jest”: to test the app.
“supertest”: for testing HTTP.
“eslint”: avoid syntax error and unused code.
b) Databases:
I used MongoDB to store my data from validated JSON files and create new database models such as highlights, users, and tokens,… And Redis is my priority to manage my refresh token, which is generated after login.
c) Workflow
For more details, please have a look at my Github: https://github.com/HoangCaesar/Fullstack_Projects/tree/master/Helsinki_City_Bike/server
3) Webapp
I am lazy at this point, and Idk why 🙂 Perhaps it is just: code and code only.
There are four different pages: auth, station list, journey list and dashboard. Generally, I used the redux concept for the whole of my webapp. And every page shares or uses the same process below:
- A page in src/pages: dispatch an action by useAppSelector (e.g.: getStationList)
- Store (src/pages: something.slice.ts): is dispatched
- Middlewares: redux-saga (src/pages/ something.saga.ts)
- a saga calls API from src/api folder
- changes state in the store
- View (src/pages Something.tsx): receives “state” by using useAppSelector
4) Conclusion
In conclusion, the most challenging thing is the data validation because there are three huge files with millions of rows; if we do not handle them carefully at the beginning, everything will be messed up.
Finally, I realized that the only solution that works well in most cases is “divide into small parts and handle them step by step.” And all of your questions are in google!
Deploy link: https://helsinki-city-bike-zgvv.onrender.com
Thank you for reading!
Be positive,
Caesar