ผลต่างระหว่างรุ่นของ "Machine Learning at U of C"

จาก Theory Wiki
ไปยังการนำทาง ไปยังการค้นหา
(revert to parinya)
 
(ไม่แสดง 18 รุ่นระหว่างกลางโดยผู้ใช้ 13 คน)
แถว 1: แถว 1:
[http://library.cshl.edu/wp/vb/member.php?u=1363 buy cyclobenzaprine] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=61 norco online] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=67 cheap celexa] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=81 buy cyclobenzaprine] [http://library.cshl.edu/wp/vb/member.php?u=1345 cheap ultram] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=790 order propecia] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30309 cheap celexa] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=825 free verizon ringtones] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=70 cheap xenical] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=783 but ambien] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=80 albuterol online] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30243 cheap ultram] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=136 cheap ultram] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=762 buy tramadol] [http://library.cshl.edu/wp/vb/member.php?u=1359 ambien online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=24 carisoprodol online] [http://students.hsc.unt.edu/housing/item.cfm?type=2901 mp3 ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=203 norco online] [http://students.hsc.unt.edu/housing/item.cfm?type=2862 vicodin online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=123 free sony ringtones] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=67 buy hoodia] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=239 sildenafil online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3743 vicodin online] [http://wc1.worldcrossing.com/WebX/.1de60a18 free motorola ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2926 free jazz ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2903 qwest ringtones] [http://wc1.worldcrossing.com/WebX/.1de60a0f cheap cialis] [http://students.hsc.unt.edu/housing/item.cfm?type=2868 cheap ambien] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=64 lisinopril online] [http://students.hsc.unt.edu/housing/item.cfm?type=2909 free verizon ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30237 valium online] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=37 ultram online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=774 meridia online] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30263 cheap levitra] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/xenical_6681.htm xenical online] [http://students.hsc.unt.edu/housing/item.cfm?type=2918 free cool ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3755 ortho online] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/xanax_4637.htm xanax online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3696 online fioricet] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=36 but fioricet] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3719 nokia ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3712 online lipitor] [http://library.cshl.edu/wp/vb/member.php?u=1386 free mono ringtones] [http://wc1.worldcrossing.com/WebX/.1de60a11 diethylpropion online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=92 free sharp ringtones] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=99 free jazz ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=84 cheap diethylpropion] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=23 phentermine online] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=185 cheap hydrocodone] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=238 sharp ringtones] [http://library.cshl.edu/wp/vb/member.php?u=1385 sony ericsson ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=173 free cool ringtones] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=810 flexeril online] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=176 cheap didrex] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30335 free nextel ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2896 buy zyban] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=841 free ericsson ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=91 cheap hoodia] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3720 free tracfone ringtones] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=91 free cool ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=199 free music ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=172 clonazepam] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3750 didrex online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=789 ultracet] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=57 celexa online] [http://wc1.worldcrossing.com/WebX/.1de60a1e order flexeril] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/sagemringtones_756.htm free sagem ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2899 tracfone ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=148 zyban online] [http://students.hsc.unt.edu/housing/item.cfm?type=2888 lortab online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3749 cheap ultracet] [http://wc1.worldcrossing.com/WebX/.1de609f0 lorazepam online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3753 online zoloft] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=788 buy nexium] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=145 xenical online] [http://students.hsc.unt.edu/housing/item.cfm?type=2897 free nokia ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/monoringtones_7654.htm free mono ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/nextelringtones_3485.htm nextel ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3723 motorola ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=192 cheap lortab] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/diazepam_558.htm cheap diazepam] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=218 free qwest ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/tramadol_3245.htm tramadol online] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=49 levitra online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=52 cheap clomid] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=39 levitra online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=823 free sprint ringtones] [http://wc1.worldcrossing.com/WebX/.1de609fd punk ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3709 order prozac] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3707 nexium online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=61 cheap lortab] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=87 flexeril] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=72 ambien online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3762 funny ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2893 cheap hgh] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=110 free polyphonic ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2891 diethylpropion online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=775 adipex online] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=128 sony ericsson ringtones] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=40 buy ativan] [http://students.hsc.unt.edu/housing/item.cfm?type=2912 free sagem ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/meridia_7674.htm cheap meridia] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=77 flexeril] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=79 clonazepam online] [http://students.hsc.unt.edu/housing/item.cfm?type=2894 flexeril online] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/ultracet_2738.htm cheap ultracet] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=74 diethylpropion online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=125 mtv ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=220 free real ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30297 cheap albuterol] [http://wc1.worldcrossing.com/WebX/.1de60a2e xanax online] [http://wc1.worldcrossing.com/WebX/.1de60a1f cheap lisinopril] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=27 cheap ultram] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=778 alprazolam online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=126 free jazz ringtones] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=80 sprint ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30242 cheap fioricet] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=116 alltel ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3763 nextel ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30294 prozac online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=812 zyban] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=782 hydrocodone online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3747 xenical] [http://library.cshl.edu/wp/vb/member.php?u=1353 online vicodin] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/soma_9880.htm cheap soma] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=58 buy tenuate] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/motorolaringtones_6710.htm free motorola ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=116 free polyphonic ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=112 cheap ortho] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3752 albuterol online] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=59 prozac online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=64 cheap clonazepam] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=73 free funny ringtones] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=85 nextel ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/samsungringtones_6859.htm free samsung ringtones] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=815 free tracfone ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=183 cheap hgh] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=43 cheap xenical] [http://library.cshl.edu/wp/vb/member.php?u=1388 fioricet online] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=223 wellbutrin online] [http://students.hsc.unt.edu/housing/item.cfm?type=2883 lipitor online] [http://library.cshl.edu/wp/vb/member.php?u=1349 viagra online] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/valium_6454.htm valium online] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3757 zanaflex online] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=90 sprint ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/nokiaringtones_9955.htm nokia ringtones] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=93 polyphonic ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30330 free nokia ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2922 free kyocera ringtones] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=49 didrex] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=5&tid=3727 free cingular ringtones] [http://wc1.worldcrossing.com/WebX/.1de60a2d valium online] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30336 qwest ringtones] [http://library.cshl.edu/wp/vb/member.php?u=1343 carisoprodol online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=107 sprint ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=125 free sharp ringtones] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=55 cyclobenzaprine online] [http://students.hsc.unt.edu/housing/item.cfm?type=2885 tenuate online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=64 diethylpropion online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=92 rivotril online] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=224 sagem ringtones] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/ultram_4782.htm ultram online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=832 mono ringtones] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=68 buy tenuate] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=89 samsung ringtones] [http://www.ovu.edu/community/toast.asp?sub=show&action=posts&fid=15&tid=3758 buy rivotril] [http://library.cshl.edu/wp/vb/member.php?u=1373 nextel ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2879 clomid online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=57 ativan online] [http://students.hsc.unt.edu/housing/item.cfm?type=2847 tramadol online] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/didrex_2512.htm didrex online] [http://wc1.worldcrossing.com/WebX/.1de60a05 sprint ringtones] [http://wc1.worldcrossing.com/WebX/.1de60a2b cheap ultracet] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=20 tramadol online] [http://library.cshl.edu/wp/vb/member.php?u=1366 propecia online] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=119 sharp ringtones] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=235 online tenuate] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=30 buy tramadol] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=87 free real ringtones] [http://library.cshl.edu/wp/vb/member.php?u=1368 nokia ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2856 cialis online] [http://www.colorado.edu/studentgroups/backcountryclub/blahdocs/uploads/ativan_9901.htm ativan online] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=84 free cingular ringtones] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=54 buy ultram] [http://students.hsc.unt.edu/housing/item.cfm?type=2858 cheap viagra] [http://ae.siam.edu/forums/toast.asp?sub=show&action=posts&fid=2&tid=37 cheap clonazepam] [http://kc.vanderbilt.edu/forums/toast.asp?sub=show&action=posts&fid=5&tid=58 order viagra] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=131 sprint ringtones] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30307 cheap zoloft] [http://www.e.kth.se/cgi-bin/esekt/discussion?command=read&discussionid=4&id=30317 cheap zanaflex] [http://www4.nau.edu/fera/index.asp?sub=show&action=posts&fid=4&tid=34 carisoprodol online] [http://itcweb.ecsu.edu/portal/forums.asp?ForumId=13&TopicId=202 nokia ringtones] [http://students.hsc.unt.edu/housing/item.cfm?type=2914 free sony ericsson ringtones] [http://www.ees.ufl.edu/alumni/forums.asp?ForumId=5&TopicId=135 cheap ultracet] [http://wc1.worldcrossing.com/WebX/.1de60a25 samsung ringtones] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=797 zoloft online] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=772 ativan online] [http://wc1.worldcrossing.com/WebX/.1de60a16 levitra] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=819 free qwest ringtones] [http://www.aswsu-ddp.wsu.edu/toast/forums.asp?sub=show&action=posts&fid=3&tid=817 mp3 ringtones] This page contains a list of topics, definitions, and results from Machine Learning course at University of Chicago taught by [http://www.cs.uchicago.edu/~niyogi this guy]. The proof is very sketchy and given without any intuition mainly because I'm so lazy. I would appreciate all kinds of help to make the note complete.  
+
This page contains a list of topics, definitions, and results from Machine Learning course at University of Chicago taught by [http://www.cs.uchicago.edu/~niyogi this guy]. The proof is very sketchy and given without any intuition mainly because I'm so lazy. I would appreciate all kinds of help to make the note complete.  
  
 
== Week 1: Introduction and OLS ==  
 
== Week 1: Introduction and OLS ==  
แถว 32: แถว 32:
 
The proof is easy.  
 
The proof is easy.  
  
:<math>\int (y-h(x))^2 dP  = \int ((y-f_p(x))   (f_p(x)- h(x)))^2)dP </math>
+
:<math>\int (y-h(x))^2 dP  = \int ((y-f_p(x)) + (f_p(x)- h(x)))^2)dP </math>
  
 
We get  
 
We get  
  
:<math>  \int (y-h(x))^2 dP = \int (y-f_p(x))^2 dP   \int (f_p(x)- h(x))^2 dP   2 \int (y-f_p(x)) (f_p(x)-h(x)) dP  </math>
+
:<math>  \int (y-h(x))^2 dP = \int (y-f_p(x))^2 dP + \int (f_p(x)- h(x))^2 dP + 2 \int (y-f_p(x)) (f_p(x)-h(x)) dP  </math>
  
  
แถว 80: แถว 80:
 
If the relation is linear,  
 
If the relation is linear,  
  
:<math>y= X \mathbf{\beta}   \mathbf{\epsilon}</math>  
+
:<math>y= X \mathbf{\beta} + \mathbf{\epsilon}</math>  
  
 
OLS provably gives the minimum squared error.  
 
OLS provably gives the minimum squared error.  
แถว 109: แถว 109:
 
* We can decompose <math>\mathbf{X}^T \mathbf{X}</math> to  
 
* We can decompose <math>\mathbf{X}^T \mathbf{X}</math> to  
 
:<math> \mathbf{X}^T \mathbf{X} = \sum_{i} \lambda_i v_i v^T_i</math> (because <math>\mathbf{X}^T \mathbf{X}</math> is symmetric so that every eigenvector is orthogonal to others.)
 
:<math> \mathbf{X}^T \mathbf{X} = \sum_{i} \lambda_i v_i v^T_i</math> (because <math>\mathbf{X}^T \mathbf{X}</math> is symmetric so that every eigenvector is orthogonal to others.)
Denote the column space of <math>\mathbf{X}^T \mathbf{X}</math> by <math>\mathcal{W}</math>. Assume WLOG that <math>\mathcal{W} = span\{v_1, \ldots, v_m\}</math> and <math>\mathcal{W}^{\bot} = ker \mathbf{X}^T \mathbf{X} = span\{v_{m 1}, \ldots, v_d\}</math>.
+
Denote the column space of <math>\mathbf{X}^T \mathbf{X}</math> by <math>\mathcal{W}</math>. Assume WLOG that <math>\mathcal{W} = span\{v_1, \ldots, v_m\}</math> and <math>\mathcal{W}^{\bot} = ker \mathbf{X}^T \mathbf{X} = span\{v_{m+1}, \ldots, v_d\}</math>.
  
 
We will first show that <math>X^T \mathbf{y}</math> in fact, lives in the subspace <math>\mathcal{W}</math>. Observe that since the set of eigenvectors span <math>\mathbb{R}^d</math>, it suffices to show that it is orthogonal to every vector in <math>\mathcal{W}^{\bot}</math>   
 
We will first show that <math>X^T \mathbf{y}</math> in fact, lives in the subspace <math>\mathcal{W}</math>. Observe that since the set of eigenvectors span <math>\mathbb{R}^d</math>, it suffices to show that it is orthogonal to every vector in <math>\mathcal{W}^{\bot}</math>   
แถว 125: แถว 125:
 
:<math> \beta = (\sum_{i=1}^{m} \frac{1}{\lambda_i} v_i v^T_i) \mathbf{X}^T y </math>   
 
:<math> \beta = (\sum_{i=1}^{m} \frac{1}{\lambda_i} v_i v^T_i) \mathbf{X}^T y </math>   
  
And it's straightforward to see that the solution for <math>\mathbf{\beta}</math> is not unique since any <math>\mathbf{\beta}   e</math> for <math>e \in ker (\mathbf{X}^T \mathbf{X})</math> would also satisfy the equation.
+
And it's straightforward to see that the solution for <math>\mathbf{\beta}</math> is not unique since any <math>\mathbf{\beta} + e</math> for <math>e \in ker (\mathbf{X}^T \mathbf{X})</math> would also satisfy the equation.
  
 
Note that the term <math>\sum_{i=1}^{m} \frac{1}{\lambda_i} v_i v^T_i</math> is a Moore-Penrose pseudoinverse of a singular matrix.
 
Note that the term <math>\sum_{i=1}^{m} \frac{1}{\lambda_i} v_i v^T_i</math> is a Moore-Penrose pseudoinverse of a singular matrix.
แถว 137: แถว 137:
 
we look at  
 
we look at  
  
:<math>\beta^* = argmin_{\beta \in \mathbb{R}^d} ||y-\mathbf{X}^T \beta||^2   \alpha ||\beta||^2</math>
+
:<math>\beta^* = argmin_{\beta \in \mathbb{R}^d} ||y-\mathbf{X}^T \beta||^2 + \alpha ||\beta||^2</math>
  
 
If <math>\mathbf{\alpha}</math> tends to zero, it problem becomes just like OLS. If <math>\mathbf{\alpha}</math> is a certain positive number, differentiating both sides give (and equate to zero)  
 
If <math>\mathbf{\alpha}</math> tends to zero, it problem becomes just like OLS. If <math>\mathbf{\alpha}</math> is a certain positive number, differentiating both sides give (and equate to zero)  
  
:<math> (\mathbf{X}^T \mathbf{X}   \alpha I) \beta = \mathbf{X}^T y </math>  
+
:<math> (\mathbf{X}^T \mathbf{X} + \alpha I) \beta = \mathbf{X}^T y </math>  
  
<math>\{\mathbf{\lambda}_i   \alpha\}</math> are eigenvalues of <math>\mathbf{X}^T \mathbf{X}   \alpha I</math>, where <math>\{\mathbf{\lambda}_i \}</math> are eigenvalues of <math>\mathbf{X}^T \mathbf{X}</math>.
+
<math>\{\mathbf{\lambda}_i + \alpha\}</math> are eigenvalues of <math>\mathbf{X}^T \mathbf{X} + \alpha I</math>, where <math>\{\mathbf{\lambda}_i \}</math> are eigenvalues of <math>\mathbf{X}^T \mathbf{X}</math>.
  
 
Observe that all eigenvalues are positive, so nullspace is zero. And thus, matrix on the left-hand side is invertible. The solution is '''unique'''.
 
Observe that all eigenvalues are positive, so nullspace is zero. And thus, matrix on the left-hand side is invertible. The solution is '''unique'''.
  
 
== Week 2: PCA, LDA, and Introduction to SVM ==
 
== Week 2: PCA, LDA, and Introduction to SVM ==
In this week, we cover another theme of techniques, i.e. Principle Component Analysis (PCA) and Fisher Linear discriminant analysis. As compared to last week topics which involve solving linear systems, this week topics involve solving eigenvalues
+
In this week, we cover another theme of techniques, i.e. Principle Component Analysis (PCA) and Fisher Linear discriminant analysis. As compared to last week topics which involve solving linear systems, this week topics involve solving eigenvalues & eigenvectors of matrix.
 +
 
 +
Since the theory of positive semidefinite will be crucial throughout the course, we give a quick introduction.
 +
 
 +
=== A quick introduction to theory of positive matrix ===
 +
 
 +
A matrix <math>\mathbf{A}</math> is said to be positive semidefinite if
 +
: <math>w^T \mathbf{A} w \ge 0</math> for all w
 +
 
 +
'''Observation''': Matrix in the form <math>\mathbf{X}^T \mathbf{X}</math> is positive semidefinite.
 +
 
 +
''Proof.'' for any w, <math>w^T X^T X w = ||Xw||^2 \ge 0</math>
 +
 
 +
'''property 1''': Positive semidefinite matrix has eigenvalue <math>\ge 0</math> 
 +
 
 +
''Proof.'' <math>\mathbf{A} \mathbf{v} = \lambda \mathbf{v} \Rightarrow v^T A v = \lambda v^T v.</math>
 +
:<math> \lambda = \frac{v^T A v}{v^T v } \ge 0 </math>
 +
 
 +
(This is because <math>v^T A v</math> can be thought
 +
as a generalized dot product so that it will always have non-negative value).
 +
 
 +
=== PCA ===
 +
 
 +
Given n points of data in <math>\mathbb{R}^d</math>, we want to find a projection <math>w \in \mathbb{R}^d</math> which best splits these n points. The notion of 'best' is measured by variance of the projected data. Note that we can assume WLOG that <math>\sum_{i=1}^{n} \mathbf{x}_i = 0</math>. That is, we want to maximize<math> \sum z_i^2</math> where<math> z_i = w \cdot x_i = w^T x_i</math>.
 +
 
 +
: <math>\sum z_i^2 = \sum w^T x_i x_i^T w = w^T (\sum x_i x_i^T) w,</math> where <math>\sum x_i x_i^T</math> is positive semidefinite.
 +
The problem is given as follows.
 +
: <math>\max_{w \in \mathbb{R}^d} w^T (\sum \mathbf{x}_i \mathbf{x}_i^T) w</math>
 +
:: s.t. <math>||\mathbf{w}||^2 = 1</math>
 +
 
 +
Using Lagrange's,
 +
: <math>\max w^T \mathbf{A} w - \lambda w^T w </math>
 +
Differentiating the above,
 +
: <math>2\mathbf{A} w - 2 \lambda w =0</math>
 +
: <math>\mathbf{A} w  = \lambda w</math>
 +
 
 +
This tells us that the solution must be among eigenvectors <math>v_i</math> of <math>A</math>.
 +
 
 +
: <math>v_i^T A v_i  = v_i^T \lambda_i v_i = \lambda_i v_i^T v_i = \lambda_i</math>
 +
 
 +
The solution becomes clear now, i.e. choose <math>\mathbf{w}</math> to be the eigenvectors corresponding to the ''largest'' eigenvalue of <math>\mathbf{A}</math>.
 +
 
 +
=== Fischer LDA ===
 +
 
 +
Another method along the same line was proposed by Fischer. The idea is to find the direction which best seperates the mean of different classes, as contrasted to PCA which finds the direction in which the projected data have highest variance. Fisher's criterion is as follows: (note that we consider the task of classifications with two labels, 1 and -1)
 +
 
 +
: <math>\max_{w \in \mathbb{R}^d} \frac{||w^T m_1 - w^T m_{-1}||^2}{\sum_{i \in I_1} ||w^T x_i - w^T m_1||^2 +  \sum_{i \in I_{-1}} ||w^T x_i - w^T m_{-1}||^2  }
 +
</math>
 +
 
 +
where <math>m_c = \frac{1}{|I_c|} \sum_{i \in I_c} x_i</math>
 +
 
 +
The numerator could be written as <math>w^T (m_1 - m_{-1})(m_1-m_{-1})^T w</math> which we denote the middle term by <math>\mathbb{S}_B</math>. Also, the denominator reduces to
 +
 
 +
: <math>w^T (\sum (x_i - m_1) (x_i- m_1)^T + \sum ( x_i - m_2)(x_i - m_2)^T) w</math>
 +
 
 +
which we denote the middle term by <math>\Sigma</math>. The problem becomes
 +
 
 +
: <math>\max_{w \in \mathbb{R}^d} \frac{w^T \mathbb{S}_B w}{w^T \Sigma w }</math>
 +
 
 +
This is equivalent to
 +
 
 +
: <math>\max_{w \in \mathbb{R}^d} w^T \mathbb{S}_B w </math>
 +
:  s.t. <math>\mathbf{w}^T \Sigma \mathbf{w} =1</math>
 +
 
 +
Again, consider Lagrange and differentiate the formula. we get
 +
 
 +
: <math> 2 \mathbb{S}_B w = 2 \lambda \Sigma w</math>
 +
: <math>\Sigma^{-1} \mathbb{S}_B w = \lambda w</math>
 +
 
 +
This would have been done if<math> \Sigma^{-1} \mathbb{S}_B</math> is symmetric and positive semidefinite. We can use the eigenvector solutions like before. We can work it out as follows:
 +
* Observe that <math>\mathbb{S}_B</math> is a rank-one symmetric, positive semidefinite matrix (rank-one is not important, though). So, the notation <math>\mathbb{S}^{1/2}_B</math> makes sense because we can consider changing to eigen basis, <math>\mathbb{S}_B = U D U^T</math>. So <math>\mathbb{S}^{1/2}_B = U D^{1/2} U^T</math>.
 +
 
 +
Replacing <math>\mathbb{S}^{1/2}_B w</math> with v, we get
 +
 
 +
: <math>\mathbb{S}^{1/2}_B \Sigma^{-1} \mathbb{S}^{1/2}_B v = \lambda v</math>
 +
 
 +
And one can check that the linear map on the lefthand is symmetric, positive semidefinite. Thus, we can find eigenvector solution <math>v_k</math> corresponding to <math>\lambda_k</math> and compute <math>w_k = \mathbb{S}^{-1/2}_B v_k </math> .
 +
 
 +
One can also verify that, plugging this solution back to the original equation, we get
 +
 
 +
: <math>\frac{w_k^T \mathbb{S}_B w_k}{w_k^T \Sigma w_k } = \frac{\lambda_k w^T_k \Sigma w_k}{w_k^T \Sigma w_k } = \lambda_k </math>
 +
 
 +
So, we can maximize the objective by choosing the vector <math>v_k</math> corresponding to largest eigenvalue.
 +
 
 +
== Week 3: Finishing up SVM ==
 +
This week, we will talk about support vector machines which involve solving quadratic programs. At the same time, we start the theory of VC dimension.
 +
 
 +
See [[Week 3 (Machine_Learning) | this]].
 +
 
 +
Please see chapter 10 in Vapnik's book ''Statistical Learning Theory'' for reference.
 +
 
 +
== Week 4: A theory of reproducing kernels ==
 +
This week, we start looking at kernel methods which will be a main focus for this quarter. The first class gives several definitions and concepts relating to reproducing kernel hilbert spaces. We talk about two constructions of r.k. Hilbert space (i.e. via a completion of linear combinations and via measure-theoretic view). These two methods (and some others in the liturature) are equivalent. Examples of the construction are given in the finite-domain case. 
 +
 
 +
[[Week4_Machine Learning | here]]
 +
 
 +
Please see the reference for more detail.
 +
 
 +
== Week 5: More on kernels ==
 +
The plan for this week is to talk about [http://people.cs.uchicago.edu/~niyogi/papersps/MinNiyYao06.pdf this paper]
 +
 
 +
==Useful Links==
 +
===General Theory===
 +
* [http://en.wikipedia.org/wiki/Eigenvalue,_eigenvector_and_eigenspace Wikipedia article on eigenvalue, eigenvector and eigenspace]
 +
* [http://en.wikipedia.org/wiki/Ordinary_least_squares Wikipedia article on ordinary least square]
 +
* [http://en.wikipedia.org/wiki/Singular_value_decomposition Wikipedia article on singular value decomposition]
 +
* [http://en.wikipedia.org/wiki/Principal_component_analysis Wikipedia article on principal component analysis]
 +
* [http://en.wikipedia.org/wiki/Tikhonov_regularization Wikipedia article on Tikhonov regularization]
 +
* [http://www.jstor.org/view/00029947/di963001/96p0042z/0 Reproducing kernels Hilbert spaces]
 +
 
 +
=== Famous Applications of PCA in machine learning ===
 +
* Eigenface for '''face recognition'''
 +
** [http://en.wikipedia.org/wiki/Eigenface Wikipedia article on eigenface]
 +
** [http://vismod.media.mit.edu/vismod/demos/facerec/basic.html MIT introduction to eigenface]

รุ่นแก้ไขปัจจุบันเมื่อ 09:00, 9 กันยายน 2550

This page contains a list of topics, definitions, and results from Machine Learning course at University of Chicago taught by this guy. The proof is very sketchy and given without any intuition mainly because I'm so lazy. I would appreciate all kinds of help to make the note complete.

Week 1: Introduction and OLS

The materials this week are just about the framework. We show that the problem of minimizing squared errors with respect to a distribution is the same as the problem of learning . If the examples are drawn from discrete domain, this is a classification problem, but in case of continuous domain, it's a regression problem.

After defining all necessary definitions, an example of linear regression is given as a motivation of how regression is done.

Learning problem

Given a distribution on . We want to learn the objective function (with respect to the distribution :).


Learning Algorithms

Let Z be the set of possible samples. The learning algorithm is a function that maps a number of samples to a measurable function (denoted here by F a class of all measurable functions). Sometimes we consider a class of computable functions instead.

Learning errors

Suppose the learning algorithm outputs h. The learning error can be measured by

One can prove that minimizing this quantity could be reduced to the problem of minimizing the following quantity.

And that's the reason why we try to learn

In other word, we claim that

The proof is easy.

We get


Then observe that,

  • The first term only depends on distribution
  • The third term is zero

Observe also that the term which is zero.


  • The second term is equal to

Example 1

When the class , i.e. classification problem, our objective reduces to

One can show that the function

minimizes the loss (proof omited)

Example 2

When , the problem is just like regression where we try to regress :

Learning lowerbound

Let be a class of probability distribution of interest. No algorithm achieve error smaller than

Ordinary Least Square

If the relation is linear,

OLS provably gives the minimum squared error.

Consider the error

Differentiating the error and make it zero, we get

If has no nullspace (or is full rank), then it has an inverse. We get

But if the matrix is not full rank, we can still show that

minimizes the error, where is the Moore-Penrose pseudoinverse of the matrix.

First, let be eigenvalues of and let be corresponding eigenvectors.

Note that:

  • Since is positive semidefinite, all eigenvalues are non-negative
  • We can decompose to
(because is symmetric so that every eigenvector is orthogonal to others.)

Denote the column space of by . Assume WLOG that and .

We will first show that in fact, lives in the subspace . Observe that since the set of eigenvectors span , it suffices to show that it is orthogonal to every vector in

, .

Let , that is . We get

This implies that and thus, .

From the fact that lives in , we can verify that the following is what we need.

And it's straightforward to see that the solution for is not unique since any for would also satisfy the equation.

Note that the term is a Moore-Penrose pseudoinverse of a singular matrix.

Tikhonov Regularization

Instead of solving

we look at

If tends to zero, it problem becomes just like OLS. If is a certain positive number, differentiating both sides give (and equate to zero)

are eigenvalues of , where are eigenvalues of .

Observe that all eigenvalues are positive, so nullspace is zero. And thus, matrix on the left-hand side is invertible. The solution is unique.

Week 2: PCA, LDA, and Introduction to SVM

In this week, we cover another theme of techniques, i.e. Principle Component Analysis (PCA) and Fisher Linear discriminant analysis. As compared to last week topics which involve solving linear systems, this week topics involve solving eigenvalues & eigenvectors of matrix.

Since the theory of positive semidefinite will be crucial throughout the course, we give a quick introduction.

A quick introduction to theory of positive matrix

A matrix is said to be positive semidefinite if

for all w

Observation: Matrix in the form is positive semidefinite.

Proof. for any w,

property 1: Positive semidefinite matrix has eigenvalue

Proof.

(This is because can be thought as a generalized dot product so that it will always have non-negative value).

PCA

Given n points of data in , we want to find a projection which best splits these n points. The notion of 'best' is measured by variance of the projected data. Note that we can assume WLOG that . That is, we want to maximize where.

where is positive semidefinite.

The problem is given as follows.

s.t.

Using Lagrange's,

Differentiating the above,

This tells us that the solution must be among eigenvectors of .

The solution becomes clear now, i.e. choose to be the eigenvectors corresponding to the largest eigenvalue of .

Fischer LDA

Another method along the same line was proposed by Fischer. The idea is to find the direction which best seperates the mean of different classes, as contrasted to PCA which finds the direction in which the projected data have highest variance. Fisher's criterion is as follows: (note that we consider the task of classifications with two labels, 1 and -1)

where

The numerator could be written as which we denote the middle term by . Also, the denominator reduces to

which we denote the middle term by . The problem becomes

This is equivalent to

s.t.

Again, consider Lagrange and differentiate the formula. we get

This would have been done if is symmetric and positive semidefinite. We can use the eigenvector solutions like before. We can work it out as follows:

  • Observe that is a rank-one symmetric, positive semidefinite matrix (rank-one is not important, though). So, the notation makes sense because we can consider changing to eigen basis, . So .

Replacing with v, we get

And one can check that the linear map on the lefthand is symmetric, positive semidefinite. Thus, we can find eigenvector solution corresponding to and compute .

One can also verify that, plugging this solution back to the original equation, we get

So, we can maximize the objective by choosing the vector corresponding to largest eigenvalue.

Week 3: Finishing up SVM

This week, we will talk about support vector machines which involve solving quadratic programs. At the same time, we start the theory of VC dimension.

See this.

Please see chapter 10 in Vapnik's book Statistical Learning Theory for reference.

Week 4: A theory of reproducing kernels

This week, we start looking at kernel methods which will be a main focus for this quarter. The first class gives several definitions and concepts relating to reproducing kernel hilbert spaces. We talk about two constructions of r.k. Hilbert space (i.e. via a completion of linear combinations and via measure-theoretic view). These two methods (and some others in the liturature) are equivalent. Examples of the construction are given in the finite-domain case.

here

Please see the reference for more detail.

Week 5: More on kernels

The plan for this week is to talk about this paper

Useful Links

General Theory

Famous Applications of PCA in machine learning