Yes, I too used to struggle with this.

Debugging

Learn how to debug. For me, it's a lifesaver for me to be able to step through some code to figure out what it actually does, instead of me trying to read the code to figure out what it may do. Yes, I do this for my own code too, because we tend to make assumptions, and I want to confirm those, always.

That means learning how to setup your IDE of choice - I presume you use vscode, so you'll then have to google for "vscode debugging". Maybe you'll have to install some addons to add the support, probably setup some launch.json in a local .vscode folder. It all depends on your language of choice.

Learn how to test. This goes great with debugging. I write code in Python, so I end up creating src/ and tests/ folders. src/ for my actual code, and tests/ for my tests. I can use either pytest on the terminal, or just the vscode test addons to run tests.

Anyway, my tests end up being something like this:

src/my_app/main.py or something, with src/my_app/__init__.py existing to turn that folder into a module:

def main():
    # some code I want to run

Then in tests/test_main.py (mirroring the src/ folder; adding test_ makes the file findable for pytest, and I call it main to make it easier to link to the main code):

from my_app import main

def test_main():
    main()

This is how I always start off with - just a simple piece of code that does a thing, and a test with almost the same name as the function I'm trying to test. I can now add a breakpoint inside test_main and run the test within vscode, which means I have a way of hooking into the main function.

Think about the process of your application

Think about how to cut up the steps to create your application into smaller and smaller steps. Whenever something feels insurmountable, I'll just have to stop in my tracks and mentally cut up a task into smaller and smaller steps, until I feel comfortable to take some steps.

I'm a data engineer, which means I tend to write code to 'ingest' data (which means, grab it from source A and put it into target B (where B is some centralized location to store all raw data).

So the main task is:

1. Write ingestion

I then have to figure out "what is the source", because that dictates how I grab the data (do I have to loop over all folders in an SFTP server? Is there a state file that makes my life easier? Do I use an API instead?)

1. Write ingestion
   1. figure out what the source is
   2. Is there an SFTP state file? is it an API?
   3. Do I need a username/password? Some API key?

I then start writing a small piece of code that connects to the source, or just grabs some random data to show the connection works.

So now I can grab some data. Is that data too large to ingest all at once? If a file is super large, I may not be able to hold it into data, which means using a buffer. And how many files are there to download? Should I batch those?

1. Write ingestion
   1. figure out what the source is | SFTP
   2. Is there an SFTP state file? is it an API? | there is a state file
   3. Do I need a username/password? Some API key? | usename/password
   4. How big are the files?
   5. How many files are there?

and this is how I slowly grow my applications from an idea "ingest all data from some source" into something that can actually run.

Now, I do have some experience and know that filesize and filecount are important to take into account, but that's something I learned along the way.

My first applications just loaded whole files into memory (a bad idea if your memory limit is 4 GB, and I'm trying to load multiple 1GB sized files into memory 😆), and taking local state (which files have I already downloaded) and external state (which one have updated or been added?) into account, etc.

Anyway, you're already on the right path: You already know a weak point, and you're smart enough to know your limits and ask for help when you're stuck. That's one of the fastest ways to grow as a programmer.