To begin with, establish your web server's rewrite rules to allow friendly URL rewriting. If you are using Apache, you can simply re-name the included ht.access file in your site root to .htaccess. If your site is being run from a subdirectory, make sure to edit that to show the subdirectory in the RewriteBase directive. Here are some examples for using other web servers such as lighttpd and nginx.
Create a new resource, name it Not Found, with an alias of not-found, and set it to be published and to not show in the menu. Assign it to have a blank template, and put something like "Page Not Found" into its Content field.
Now, in the Manager, go to System -> System Settings. In the Filter by area drop-down, select "Site". Change the error_page setting to reflect the resource ID of that "Not Found" resource we created earlier - it should be 2. Simply double-click on the Value field, edit, then either hit Enter or click outside of the input field. A little red triangle will appear to indicate a change in the field, then it will disappear as the new value is automatically saved.
Scroll down to the "Site name" row, and edit the site name in the same way.
Now in the Filter by area drop-down, select "Friendly URL". Scroll down and change the Use Friendly URL to yes. In your browers, try using the friendly URLs to access your not-found.html page and back to the index.html page. If this isn't working, check your web server's rewrite rules.