I made a simple melody in FL studio that was in C Major. The melody consisted of two phrases both of which started on D. The only difference between the two is the second phrase ended on C (the tonic of C major). The only problem is, it didn’t feel resolved ending on the tonic like I thought it would. When I changed the last note of the tonic to a D it sounded resolved even though D is not the tonic of C Major. After looking further into it, I realized that all the notes I used were also in D Dorian as well as C Major. So, my question is what defines a melody as being in a certain key to the listener? What makes the listener want to hear notes in a certain scale within a melody or resolve on the tonic of a certain scale?
It can partially be subjective. Ears want to hear what they expect. If it’s just a melody in a vacuum then it’s possible that it’s just what was in your head and someone else wouldn’t have been content with where you ended. Some notes like the 7th are less likely to feel like familiar resting place for the ear tho. Once you throw backing chords or other notes into the equation everything starts to complicate.
As for rules dictating what notes create tension and what notes resolve said tension yadda yadda. Music is more like well worn roads between notes. Sometimes the more interesting roads are ones that people don’t travel as often. See giant steps by John Coltrane. He just goes wherever and bends the music into submission.
Tertian harmony. The third of a chord and in a key (mediant) determine the key. The sixth (sub-mediant) hints at it.
Now for secudal or quartal harmony, well…
I can’t really answer your question I’m afraid as my understanding of theory is limited but for the sake of curiosity and testing how about you post both versions of the mentioned melody and see how other people react to it. Maybe other people will feel differently about the resolution on C or D than you do, which might reveal insight into how you perceive melody subjectively.