Why is the final syllable of missle and hassle spelled -sle while the final syllable of castle is spelled -stle?
Back in the 14th through the 16th centuries, there was a strong tendency to simplify the pronunciation of certain strings of three consonants, usually by not pronouncing the middle one, without changing the spelling. This was very common in the string [stl], so we have the following words like castle: apostle, bristle, bustle, epistle, gristle, hustle, jostle, mistle, nestle, rustle, thistle, throstle, trestle, whistle, wrestle.
Notice that the [t] is pronounced in words ending <stel>, because there is no string of three consonants in <stel>: canistel, hostel, listel, pastel (with the <e> stressed). Related to these words is pestle, which has two accepted pronunciations – one older one with the [t] pronounced, one newer one without. The reason for these variants appears to be that earlier the word was spelled <pestel>, which, with no three-consonant string, would retain the [t]. But its spelling changed to <stle> in the 15th century, which encouraged newer pronunciations with no [t] sound.
A similar simplification occurs in consonant strings in which the letters <st> are followed by the sound [n], either spelled <n>, as in chestnut and mustn’t, or spelled with an <e> that spells schwa followed by <n>, as in chasten, christen, fasten, glisten, hasten, listen, moisten. The American Heritage Dictionary treats such words as ending with a full schwa plus [n]; Merriam-Webster dictionaries treat them as ending with a syllabic [n] with no preceding schwa. (A syllabic consonant is one that can form a syllable by itself with no vowel sound.) In the Merriam-Webster treatment all the words in this paragraph are of the same sound pattern, though involving two different spellings.
Notice that in [stn] strings in which the [n] is not syllabic, the [t] is pronounced: augustness, bastnaesite, earnestness, postnasal, steadfastness, vastness.
So the answer to your question is that though the spelling of words like missle is pretty straightforward, the spelling of words like castle is complicated by a centuries old simplification of a string of three consonant sounds to just two by eliminating the middle [t] sound, without changing the spelling.