No, you won’t lose your work visa if you quit your agency in Japan. The Japanese executive owns and is responsible in your visa, not your agency. If your company threatens you and your visa status, remember to contact hello work immediately. You will must find an alternative job and inform immigration that you’ve got changed companies. You can do this with a simple form so that you can mail in.
Japanese immigration specialists change this type frequently so contact your local immigration office for particulars and an up-to-the-minute form. The company you are leaving is legally required to give you a form called a 退職証明書 taishoku shoumei sho, that is an evidence that you’ve resigned from a company. You’ll want to keep this kind and current it in the event you renew your visa.